Fairness and Bias

Fairness is a social rather than a psychometric concept. Its definition depends on what one considers to be fair. Fairness has no single meaning and, therefore, no single definition, whether statistical, psychometric, or social. The Standards notes four possible meanings of “fairness.” The first meaning views fairness as requiring equal group outcomes (e.g., equal passing rates for subgroups of interest). The Standards rejects this definition, noting that it has been almost entirely repudiated in the professional testing literature. It notes that while group differences should trigger heightened scrutiny for possible sources of bias (i.e., a systematic error that differentially affects the performance of different groups of test takers), outcome differences in and of themselves do not indicate bias. It further notes that there is broad agreement that examinees with equal standing on the construct of interest should, on average, earn the same score regardless of group membership. The second meaning views fairness in terms of the equitable treatment of all examinees. Equitable treatment in terms of testing conditions, access to practice materials, performance feedback, retest opportunities, and other features of test administration, including providing reasonable accommodation for test takers with disabilities when appropriate, are important aspects of fairness under this perspective. There is consensus on a need for equitable treatment in test administration (although not necessarily on what constitutes equitable treatment). The third meaning views fairness as requiring that examinees have a comparable opportunity to learn the subject matter covered by the test. However, the Standards notes that this perspective is most prevalent in the domain of educational achievement testing and that opportunity to learn ordinarily plays no role in determining the fairness of employee selection procedures. One exception would be settings where the organization using the tests purposely limits access to information needed to perform well on the tests on the basis of group membership. In such cases, while the test itself may be unbiased in its coverage of job content, the use of the test would be viewed as unfair under this perspective. The fourth meaning views fairness as a lack of predictive bias. This perspective views predictor use as fair if a common regression line can be used to describe the predictor-criterion relationship for all subgroups of interest; subgroup differences in regression slopes or intercepts signal predictive bias. There is broad scientific agreement on this definition of predictive bias, but there is no similar broad agreement that the lack of predictive bias can be equated with fairness. For example, a selection system might exhibit no predictive bias by race or gender, but still be viewed as unfair if equitable treatment (e.g., access to practice materials) was not provided to all examinees.